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Preface 


The  current  Robotics  Collaborative  Technology  Alliance  (RCTA)  has,  as  an  integral  part  of  its 
program  plan,  the  requirement  to  perform  periodic  assessments  on  maturing  technologies.  This 
report  documents  the  planning  process  for  an  initial  baseline  assessment  and  presents  detailed 
guidelines  for  experiment  conduct.  The  challenging  and  complex  research  areas  of  perception, 
intelligence,  human-robot  interaction,  and  dexterous  manipulation  and  unique  mobility  provide  a 
commensurate  challenge  in  the  assessment  of  their  performance.  This  assessment  challenge  was 
even  more  acute  in  that  it  represented  the  first  assessment  of  the  program  and  involved  platforms 
and  mission  considerations  different  than  those  of  the  previous  RCTA.  The  snapshot  of  that 
process  presented  here  is  in  the  form  of  a  working  document  produced  toward  the  end  of  the 
planning  cycle.  Its  value  to  the  RCTA  community  is  in  serving  as  an  example  of  how  the  RCTA 
will  approach  assessments  in  terms  of  formal  experimentation.  The  motivation  for  releasing  it  as 
a  working  document  is  to  provide  greater  insight  to  the  development  of  an  experiment  than  is 
customarily  included  in  a  report  documenting  the  results  of  the  experiment.  The  intent  is  that 
this  will  be  helpful  to  RCTA  members  as  they  plan  future  assessments  of  RCTA  technologies. 

Because  this  was  a  working  document,  the  actual  experiment  details  differed  in  some  ways  from 
this  initial  plan,  even  though  the  general  strategy  remained  intact.  For  example,  the  side-slope 
approach  to  a  drop-off  was  abandoned  at  the  test  site  because  an  important,  limiting  feature  of 
the  intended  course  segment  was  not  discemable  from  the  aerial  photograph  used  in  planning  but 
was  immediately  obvious  when  viewed  in  person.  Other  differences  are  present,  but  we  will 
leave  those  discussions  to  the  final  report  for  the  experiment.  In  general,  however,  the  details  of 
the  experiment  outlined  here  were  implemented. 
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1.  Purpose 


This  report  outlines  the  2011  baseline  assessment  experimental  strategy  to  be  conducted  at  Fort 
Indiantown  Gap  (FTIG),  PA,  in  the  Combined  Arms  Collective  Training  Facility  (CACTF)  1-3 
August.  The  purpose  of  the  baseline  assessment  is  to  initiate  the  experimental  component 
integral  to  the  Robotics  Collaborative  Technology  Alliance  (RCTA)  and  focus  on  behavior 
primitives  necessary  to  achieving  autonomous  “look”  and  “move” — two  of  the  RCTA  vision 
components  of  “think,”  “look,”  “move,”  “talk,”  and  “work”  (TLMTW)  identified  in  the  FY  201 1 
Annual  Program  Plan  (APP). 


2.  Scope 


This  report  (1)  identifies  and  gives  the  rationale  for  individual  tests  within  the  overall 
experiment,  (2)  provides  a  description  of  each  test,  (3)  suggests  the  analysis  to  be  performed,  and 
(4)  notes  procedural  and  logistical  issues  to  be  addressed.  This  report  does  not  constitute  the 
detailed  test  plan/sequence,  which  will  follow  after  the  strategy  outlined  within  is  agreed  upon. 

It  should,  however,  provide  sufficient  detail  that  developers  can  plan  for  the  experiment. 


3.  Objective 


This  report  synthesizes  the  discussions,  demonstrations,  visitations,  and  explorations  over  the 
past  few  months  and  molds  them  into  a  sensible  starting  point  for  RCTA  experimentation.  A 
sufficient  level  of  detail  is  provided  on  the  rationale  and  conduct  of  each  test  to  provide  a  basis 
upon  which  the  community  can  accept,  decline,  or  accept  with  modification  each  test  within  the 
overall  initial  experimental  strategy. 


4.  Background 


The  APP  submitted  in  March  2011  reaffirms  the  RCTA  intent  to  include  frequent 
experimentation  on  developing  technologies.  RCTA  leadership  stressed  the  need  to  begin  that 
process  early  in  the  RCTA,  with  the  understanding  that  in  this  initial  year  technology 
development  was  just  beginning.  Additional  direction  suggested  the  initial  experimental  focus 
be  on  small  unmanned  ground  vehicle  (SUGV)  performance  within  the  TLMTW  vision. 
Visitations,  notably  with  the  20th  Support  Command  at  Aberdeen  Proving  Ground,  discussions 
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within  the  RCTA,  and  explorations  using  the  Robotic  Research-fitted  Talon  helped  forge  notions 
of  primitive  behaviors  deemed  important  to  the  TLMTW  vision  and  raise  genuine  questions  on 
performance  capabilities,  the  pursuit  of  which  would  contribute  to  RCTA  research. 


5.  Experimental  Site 


The  site  of  the  experiment  is  the  CACTF  at  FTIG.  The  CACTF  offers  an  urban  terrain  consistent 
with  a  mission  profile  envisioned  for  an  SUGV  and  many  fixed  structural  components  that  serve 
to  stress  behavior  primitives. 

Other  sites  were  considered:  the  U.S.  Army  Research  Laboratory/Human  Research  and 
Engineering  Directorate  (ARL/HRED),  ARL/Vehicle  Technology  Directorate  (VTD),  U.S. 

Army  Aberdeen  Test  Center  (ATC),  and  National  Institute  of  Standards  and  Technology 
(NIST)/Nike.  ARL/HRED  has  sufficient  ground  to  support  an  experiment  but  has  limited 
existing  course  structure  to  leverage,  and  scheduling  was  a  problem.  ARL/VTD  has  sufficient 
ground  to  support  an  experiment  but  has  no  existing  course  structure.  Also,  potential  hurdles 
associated  with  gaining  approval  for  testing  behind  the  fence  in  the  time  window  available  made 
it  impracticable.  The  ATC  site  is  an  SUGV  course  constructed  to  test  SUGV  mobility  in 
operator-controlled  environments,  but  its  configuration  does  not  support  the  range  of 
autonomous  behaviors  to  be  considered.  The  NIST/Nike  site  was  used  by  the  previous  RCTA 
and  offers  an  adequate  test  location.  Logistical  preference  and  urban  terrain  backdrop  favored 
the  CACTF  for  this  initial  investigation. 


6.  Platform/Configuration 


The  platform  will  be  the  Robotic  Research-fitted  Talon,  with  a  fixed  baseline  configuration 
(hardware/software)  for  the  experiment.  To  the  extent  possible,  parameters  will  be  fixed 
throughout  the  nine  experimental  vignettes  identified  in  this  report;  that  is,  avoid  fine  tuning  for  a 
specific  vignette  within  the  range  of  nine  challenges  to  preserve  the  sense  that  the  collective  of 
behavior  primitives  could  be  robustly  achieved.  (Absolutely  avoid  changing  parameter  settings 
within  a  vignette  according  to  different  factor  levels  to  prevent  confounded  results.)  However, 
across  vignettes  the  desire  for  robustness  must  be  balanced  with  the  need  to  collect  meaningful 
data;  a  parameter  setting  that  virtually  guarantees  failure  for  all  runs  within  a  vignette 
demonstrates  only  that  the  space  for  experimentation  (parameter  settings  and  challenge 
conditions)  was  poorly  chosen.  The  experimental  space  should  span  easy-to-hard  (or 
impossible),  so  that  something  can  be  learned.  Consequently,  we  recognize  that  to  ensure 
meaningful  data  is  collected,  certain  parameters  may  have  to  be  adjusted  according  to  the 
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challenge  of  the  vignette.  Ultimately,  a  goal  of  the  program  is  to  autonomously  make  necessary 
parameter  adjustments. 

As  an  artifact  of  the  experimental  design,  some  parameters  may  be  adjusted  to  channel  the  robot 
down  a  path  to  ensure  it  interacts  with  a  challenge. 

The  platform  is  to  be  moving  forward  in  all  runs  of  this  experiment. 


7.  Challenges  Addressed 


Many  mobility/perception  challenges  and  SUGV  tasks  linked  to  envisioned  operations  have  been 
discussed  in  the  RCTA  community.  At  this  early  stage  of  technology  development,  there  is  no 
need  to  associate  challenges  or  tasks  within  a  specific  scenario  context;  rather,  we  should 
examine  performance  with  an  eye  toward  general  applicability.  For  this  discussion,  we  draw  soft 
distinctions  between  inherent  mobility  of  the  platform,  autonomous  mobility,  and  intelligent 
control  or  higher-level  planning  activities. 

7.1  Surfaces 

Traversal  over  different  surfaces — climbing  stairs,  negotiating  ditches,  etc. — is  an  inherent 
mobility  concern.  The  platform  capability  of  moving  over  pavement,  concrete,  gravel,  stone, 
grass,  mud,  etc.,  can  be  explored  in  an  operator-controlled  setting  and  is  not  within  the  scope  of 
this  experiment.  However,  surfaces,  combined  with  slopes,  could  also  induce  slippage,  affecting 
the  NAV  solution,  or  present  challenges  to  perception  depending  on  the  coarseness  of  the  surface 
(or  in  the  case  of  grass,  the  height  of  perceived  obstacles).  In  that  sense,  surfaces  present  an 
autonomous  mobility  challenge.  To  detect  slippage,  real  time  would  require  a  ground  truth 
capability  that  we  will  not  have  for  this  experiment;  however,  a  differential  GPS  reading  will  be 
taken  at  the  beginning  and  end  of  each  run  and  compared  with  the  Talon  NAV  solution.  The 
perception  challenge  may  be  indirectly  assessed  by  measuring  speed  over  course  segments  of 
available  surfaces. 

7.2  Structure 

Common  fixed  structures  and  free  obstacles  must  be  negotiated  by  an  autonomous  SUGV. 
Specific  to  an  urban  setting,  there  are,  for  example,  cones,  vehicles,  trash  barrels,  doorways, 
stairs,  slopes,  drop-offs,  and  narrow  alleys.  These  challenges  are  immediate  impediments  to  the 
progress  of  the  SUGV,  requiring  it  to  negotiate  around  them,  through  them,  or  over  them.  Other 
impediments  to  progress  might  be  dynamic,  such  as  with  people  or  vehicles.  The  challenge  with 
movers  is  for  the  SUGV  to  appropriately  update  the  map  to  account  for  position  changes  and  to 
incorporate  the  dynamic  obstacles  in  its  route  planning. 
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7.3  Situations 

The  overcoming  of  immediate  obstacles  does  not  have  to  drastically  change  the  route  of  the 
SUGV;  it  overcomes  or  bypasses  the  obstacle  and  continues.  However,  certain  situations  create 
major  route  disruptions  that  require  the  SUGV  to  effectively  establish  a  new  route  segment  over 
a  portion  of  the  initial  route  or  provide  constraints  on  how  the  route  is  executed.  These 
challenges  suggest  a  need  for  higher-level  intelligent  control.  For  example,  outside  cul  de  sacs 
or  dead  ends  within  buildings  require  a  substantive  departure  from  the  initial  route.  A  maze  of 
street  clutter  might  have  the  same  effect,  as  would  interaction  with  buildings.  Further,  the  SUGV 
may  seek  to  minimize  line  of  sight  to  an  enemy  location  or  prefer  a  surface  (e.g.,  sidewalks) 
when  operating  in  a  busy  urban  setting.  Additional  constraints  provide  a  challenge  to  perception 
and  planning. 

7.4  Tasks 

The  SUGV  will  have  certain  tasks  to  perform  on  its  route.  It  could  be  carrying  tools  or  supplies, 
identifying  key  objects,  interacting  with  those  objects,  performing  recon  on  an  area,  etc.  At  this 
stage  in  the  program,  the  SUGV  available  has  a  capability  for  mapping  that  can  be  used  toward 
the  “look”  component  of  TLMTW.  The  SUGV  also  has  the  capability  to  record  video  as  it 
passes  through  an  area,  but  there  are  no  perception  algorithms  on  board  currently  to  dissect  that 
information  into  objects  of  interest  (OOIs).  For  example,  in  an  urban  setting  we  would  expect 
the  SUGV  to  perceive  people  in  various  settings:  moving,  stationary,  prone,  crouched,  standing, 
partially  obscured,  etc.  Other  OOIs  could  also  be  targeted,  for  example,  cars,  furniture,  walls, 
barrels,  windows,  and  suspended  objects.  In  this  initial  experiment,  a  baseline  will  be 
established  by  having  analysts  interpret  the  video.  As  perception  capabilities  come  online,  the 
completely  automated  parsing  of  OOI  can  be  compared. 

7.5  Summary  of  Challenges 

A  summary  of  the  challenges  considered  is  listed  as  follows: 

A.  Surfaces 

1 .  Pavement 

2.  Gravel 

3.  Grass  (various  heights) 

B.  Structure 

1 .  Doorway 

2.  Slope 

3.  Stairs 
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4.  Drop-offs 

5.  Alley 

6.  People 

C.  Situations 

1 .  Cul  de  sac,  dead  end,  maze 

2.  Street  clutter 

3.  Room  clutter 

4.  Building  exterior/interior 

D.  Tasks 

1 .  Mapping 

2.  OOI  video 


8.  Experiment 


The  experiment  is  parsed  into  a  surfaces  study  and  then  a  series  of  vignettes  to  address  the 
challenges  listed  in  section  7.5.  Each  is  presented  in  outline  form  with  regard  to  goal,  location, 
conduct,  measurement,  and  analysis.  Test  conduct  will  include  setup,  execution,  and  special 
considerations,  such  as  test  apparatus  or  obstacles. 

A.  Surfaces  Study:  Surfaces  will  be  examined  in  a  two-factor  4x4  design  with  surface 

acting  as  a  block. 

1 .  Goals:  To  establish  baselines  on  speed  over  surfaces  and  different  horizons  allowed 
by  autonomous  mobility,  especially  perception,  and  assess  the  quality  of  the  route 
taken. 

2.  Factors 

a.  Surface  (pavement,  gravel,  low  grass,  high  grass) 

b.  Horizon  (uphill,  downhill,  side  hill,  level) 

3.  Conduct:  Send  the  SIJGV  on  an  A/B  route  over  nominally  50-75  ft. 

4.  Responses 
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a.  Time  will  be  computed  from  when  the  treads  start  moving  to  when  the  robot 
reaches  the  end  point. 

b.  A/N  path  ratio  will  be  taken  as  the  ratio  of  the  actual  (A)  path  length  to  the 
nominal  (N)  straight  line,  A/B  path  length.  (It  is  important  to  note  that  there  is 
no  implied  expectation  that  the  SUGV  will  travel  in  perfect  linear  fashion  even 
in  an  open  course;  however,  this  measure  may  provide  some  additional  insight  to 
developers  on  how  route  costs  are  influenced  by  various  surface  conditions.) 

c.  Displacement  will  be  taken  as  the  difference  between  the  NAV  solution  at  end 
point  and  a  differential  GPS  solution.  Responses  will  be  based  on  the  same 
distance  A/B  route  if  possible,  but  normalized  to  the  distance  possible  for 
different  locations  if  necessary. 

5.  Replications:  (2) 

6.  Randomization:  Refers  to  order  of  experimentation.  TBD. 

7.  Locations:  Various  locations  are  required  throughout  the  CACTF.  The  exact  runs 
are  TBD. 

a.  Pavement 

(1)  Level:  in  front  of  police  station 

(2)  Uphill:  in  between  police  station  and  hotel 

(3)  Downhill:  in  between  police  station  and  hotel 

(4)  Sidehill:  in  between  police  station  and  hotel 

b.  Gravel 

(1)  South  of  two-story  office  building 

(2)  Uphill:  southwest  approach  to  church  intersection 

(3)  Downhill:  southwest  approach  to  church  intersection 

(4)  Sidehill:  southwest  approach  to  church  intersection 

c.  Low  grass 

(1)  Level:  across  the  circle 

(2)  Uphill:  next  to  southwest  approach  to  church  intersection 

(3)  Downhill:  next  to  southwest  approach  to  church  intersection 

(4)  Sidehill:  courtyard  next  to  southwest  approach  to  church  intersection 
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d.  High  grass 


(1)  Level:  southern  (upper)  end  of  courtyard 

(2)  Uphill:  courtyard 

(3)  Downhill:  courtyard 

(4)  Sidehill:  courtyard 

8.  Considerations 

a.  Parameter  settings  (cell  size,  corridor)  should  be  held  constant. 

b.  Course  preparation  for  grass  runs.  Grassy  areas  may  contain  solid  obstacles, 
such  as  rocks,  bricks,  cement  blocks,  etc. 

c.  An  adequate  run  length  must  be  established  to  account  for  location  tolerances  at 
the  end  of  the  run.  Side  hill  runs  will  be  reduced  to  one  rep. 

9.  Analysis 

a.  Supports  estimation  of  speeds  and  displacement  for  a  variety  of  surfaces. 

b.  Supports  investigation  of  differences  among  surfaces  using  Analysis  of  Variance 
(ANOVA). 

B.  Structure 

1.  Drop-off  or  negative  obstacle:  Negative  obstacles  can  be  explored  in  a  three-factor 
3x3x2  factorial  design,  with  location  serving  as  a  blocking  factor. 

a.  Goal:  To  establish  a  baseline  for  the  drop-off  the  SUGV  will  attempt  to 
negotiate. 

b.  Factors 

(1)  Start  surface  (level  concrete,  level  grass,  side  hill  grass) 

(2)  Drop  height  (high  [15  in],  medium  [10  in],  low  [5  in]) 

(3)  Landing  surface  (plywood,  sod  on  plywood) 

c.  Conduct:  Send  the  SUGV  on  an  A/B  route  with  a  tight  corridor  parameter 
setting  through  a  scheduled  drop-off  to  the  B  waypoint  beyond  the  challenge.  A 
line  will  be  attached  to  the  SUGV  to  prevent  it  from  incurring  damage  on  the 
most  difficult  drops.  Record  GPS  location  at  end  of  run. 

d.  Responses 
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(1)  Run  time  will  be  recorded  as  the  time  from  when  the  SUGV  starts  moving 
to  when  it  stops  for  “no  plan,”  remains  inactive  for  30  s,  or  clears  the 
challenge  and  reaches  the  goal. 

(2)  Decision  will  be  recorded  as  attempted  or  not  attempted  to  step  down. 

(3)  Outcome  will  be  recorded  in  two  parts:  as  a  data-collector  decision  on  the 
level  of  success  of  the  movement  past  the  challenge  and,  conditional  on 
getting  past  the  drop,  if  it  reached  the  goal. 

(4)  Displacement  is  the  difference  between  the  NAV-indicated  location  and 
GPS  location  at  the  end  of  the  run. 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Locations:  Business  1  across  from  the  townhouse  toward  the  cemetery  (level 
concrete  porch),  two-story  office  building  (side  hill  grass),  police  station  (grass 
to  retaining  wall  drop-off). 

h.  Considerations 

(1)  Need  a  landing  platform  with  adjustable  height.  I  suggest  two  half-sheets 
of  7/16-in  plywood  joined  by  two  or  three  door  hinges  along  with  a  number 
of  bricks.  We  would  rest  the  landing  on  bricks  to  a  predetermined  height 
and  let  the  other  1/2  sheet  of  plywood  ramp  to  the  ground  for  flat  landing 
runs. 

(2)  Need  a  few  pieces  of  sod  to  provide  a  different  surface  for  the  landing. 

(3)  Either  corridor  setting  or  physical  barriers  (plywood)  will  be  needed  to 
force  Talon  to  interact  with  the  drop-off  for  the  side  hill  location  and  police 
station  retaining  wall. 

i.  Analysis  will  be  descriptive/exploratory.  Where  appropriate,  ANOVA  will  be 
performed. 

2.  Stairs:  Stairs  can  be  explored  in  a  four-factor  3  x  3  x  2  x  2  design,  not  run  in  a 

factorial  manner. 

a.  Goal:  To  establish  a  baseline  for  the  height  and  type  of  stairs  the  SUGV  is 
willing  to  climb. 

b.  Factors:  Treatment  combinations  will  be  formed  from  these  four  factors,  but  not 
in  a  factorial  structure.  We  are  trying  to  take  into  account  the  -3 4 -in  length  of 
the  Talon  and  the  ~  10-in  tread  height. 
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(1)  Three  riser  heights  (12,  8,  and  4  in) 

(2)  Tread  width  (96,  24,  12,  and  8  in) 

(3)  Riser  could  be  open  (as  in  a  fire  escape)  or  closed  like  concrete  steps 

(4)  No.  of  steps  (one  or  two) 

(5)  Combinations  have  been  chosen  (1)  to  facilitate  modular  construction  using 
1/2  plywood  and  standard  framing  lumber  to  achieve  nominal  heights 
/widths  and  (2)  to  challenge  the  robot  in  realistic  stair  conditions  for 
humans.  Notation  (riser  height  x  tread  width)  for  the  one-step  case  and 
(riser  height  x  tread  width  x  riser  height  x  tread  width)  for  the  two-step 
case.  Run  each  combination  with  open  risers  and  closed  risers. 
Combinations  in  inches  are  as  follows:  (4  x  96),  (8  x  96),  (12  x  96),  (4  x8 
x  4  x  96),  (4  x  12  x  4  x  96),  (4  x  24  x  4  x  96),  (8  x  8  x  8  x  96),  (8  x  12  x  8 
x  96),  (8  x  24  x  8  x  96),  (12  x  12  x  12  x  96),  (12  x  24  x  12  x  96). 

c.  Conduct:  Send  the  robot  on  a  path  with  a  tight  corridor  parameter  to  engage 

steps. 

d.  Responses 

(1)  Run  time  will  be  recorded  as  the  time  from  when  the  SUGV  starts  moving 
to  when  it  stops  for  “no  plan,”  negotiates  the  climb,  or  remains  inactive  for 
30  s  upon  reaching  the  challenge. 

(2)  Decision  will  be  recorded  as  attempted  or  not  attempted  to  climb. 

(3)  Outcome  will  be  recorded  as  a  data-collector  decision  on  the  level  of 
success  of  the  movement  past  the  challenge. 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Location:  Bam  (constructed  steps)  and  townhouse  opportunities  as  appropriate. 

h.  Considerations 

(1)  We  may  be  able  to  vary  the  riser  height  (within  existing  steps)  by 

considering  fractions  of  the  riser  using  framing  lumber  (4  x  4,  2  x  4,  2  x  2, 
etc.)  to  soften  the  perceived  climb.  The  softening  would  impact  both  the 
riser  and  the  tread. 
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(2)  It  is  likely  that  bricks  or  block  support  of  plywood  landings  and  treads  with 
some  combination  of  lumber  may  be  required  to  completely  simulate  steps 
for  some  conditions,  but  the  townhouse  steps  should  be  incorporated  if 
feasible. 

(3)  Bricks  on  the  end  of  the  steps,  with  2x4  spanning,  might  be  able  to  give 
the  effect  of  open  risers. 

i.  Analysis  will  be  descriptive/exploratory. 

3.  Doorway:  Doorways  can  be  explored  in  a  two-factor  3x3x2  factorial  design. 

a.  Goal:  To  establish  a  baseline  for  the  opening  and  angle  of  approach  needed  to 

pass  through  a  doorway. 

b.  Factors:  Factors  take  into  account  the  -22-in  platform  width. 

(1)  Three  common  door  widths:  24,  30,  and  36  in. 

(2)  Angle  of  approach  90°,  45°,  and  0°  from  opening. 

(3)  Door  ajar  away  from  approach  in  degrees  (180  and  60).  (Note:  The  24-in 
door  at  60°  would  require  the  SUGV  to  nudge  the  door  to  pass  through.) 

c.  Conduct:  The  robot  travels  an  A/B  route  through  a  door. 

d.  Responses 

(1)  Run  time  will  be  recorded  as  the  time  from  when  the  SUGV  starts  moving 
to  when  it  stops  for  “no  plan,”  negotiates  the  door  and  fully  enters  the 
room,  or  remains  inactive  for  30  s. 

(2)  Decision  will  be  recorded  as  attempted  or  not  attempted  to  pass  through. 

(3)  Outcome  will  be  recorded  as  a  data-collector  decision  on  the  level  of 
success  of  the  movement  past  the  challenge. 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Location:  Entering  the  police  station  from  the  outside  patio. 

h.  Considerations 

(1)  Use  one  1/2  sheet  of  plywood  with  right-angle  support  to  make  it 

perpendicular  to  the  floor/ground.  Slide  it  as  we  wish  to  vary  the  width  of 
the  door  opening. 
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(2)  An  excursion  might  be  to  have  the  door  only  partially  open,  much  less  than 
60°,  with  one  of  the  wider  door  openings. 

(3)  Angles  considered  are  90°  from  straight  through  the  doorway,  45°,  or  0°. 
Intermediate  angles  could  be  considered  also,  but  would  be  excursions. 

i.  Analysis  will  be  descriptive/exploratory. 

4.  Alley:  A  narrow  passageway  can  be  explored  in  a  single-factor  design  at  three 
levels. 

a.  Goal:  To  establish  a  baseline  for  time  to  pass  through  a  narrow  passage. 

b.  Factor:  Passage  width  (full  alley,  tight  to  robot,  in  between). 

c.  Conduct:  Send  the  robot  on  an  A/B  route  along  a  narrow  passage. 

d.  Responses 

(1)  Time  to  achieve  route  will  be  recorded  as  the  time  from  when  the  SUGV 
starts  moving  to  when  it  stops  for  “no  plan,”  reaches  the  B  waypoint,  or 
remains  inactive  for  30  s. 

(2)  Outcome  will  be  recorded  as  a  data-collector  decision  on  the  level  of 
success  of  the  movement  past  the  challenge. 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Location:  Alley  north  side  of  police  station 

h.  Considerations 

(1)  Need  plywood  to  narrow  alley. 

(2)  Cell  size  fixed  or  varied  in  a  structured  manner  as  part  of  the  experiment. 

i.  Analysis  will  be  descriptive/exploratory. 

5.  People:  Dynamic  obstacles  on  the  map  to  be  explored  in  a  two-factor  5><2  factorial 
design. 

a.  Goal:  To  establish  a  baseline  for  whether  persistence  of  dynamic  obstacle  tracks 
on  the  map  impede  robot  progress. 

b.  Factors 

(1)  Dynamic  obstacle  path  (front  [perpendicular],  front  [parallel],  side 
[perpendicular],  side  [parallel],  orbit) 
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(2)  Speed  of  obstacle  (walk,  jog) 

(3)  Proximity  (3-4  m) 

c.  Conduct:  Have  the  robot  proceed  on  an  A/B  route  with  no  dynamic  obstacle. 

Or,  start  a  dynamic  obstacle  in  front,  beside,  or  around  the  robot  in  close 

proximity. 

d.  Responses 

(1)  Time  to  achieve  route  will  be  recorded  as  the  time  from  when  the  SUGV 
starts  moving  to  when  it  stops  for  “no  plan,”  reaches  the  B  waypoint,  or 
remains  inactive  for  30  s. 

(2)  Outcome  will  be  recorded  as  a  data-collector  decision  on  the  level  of 
success  of  the  movement  past  the  challenge. 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Location:  Street  in  front  of  the  townhouses 

h.  Considerations 

(1)  Appropriate  “proximity”  to  interact  with  route  should  be  inside  and  outside 
planning  horizon  of  5  m. 

(2)  Consider  capturing  a  decay  time  of  the  mover  on  the  map,  but  that  is 
probably  a  setting. 

(3)  Orbit  path  will  be  in  the  same  direction  as  the  sensor  rotation. 

i.  Analysis  will  be  descriptive/exploratory. 

C.  Situations/Tasks 

1 .  Cul  de  sac/dead  end:  A  cul  de  sac  and  other  major  route  disruptions  can  be  explored 

in  a  two-factor  3x3  factorial  design. 

a.  Goal:  To  establish  a  baseline  for  how  well  route  adjustments  are  implemented  to 

overcome  major  route  disruptions. 

b.  Factors 

(1)  Disruptions  (cemetery  cul  de  sac,  the  two-story  office  building  next  to  the 
small  alley,  and  a  cone  maze  in  the  courtyard) 

(2)  Severity  adjusts  route  to  reflect  (full,  moderate,  slight)  interaction  with 
disruption 
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c.  Conduct:  Have  the  robot  proceed  on  an  A/B  route  to  the  obstacle. 

d.  Responses 

(1)  Time  to  achieve  route  will  be  recorded  as  the  time  from  when  the  SUGV 
starts  moving  to  when  it  stops  for  “no  plan,”  reaches  the  B  waypoint,  or 
remains  inactive  for  30  s. 

(2)  Outcome  will  be  recorded  as  a  data-collector  decision  on  the  level  of 
success  of  the  movement  past  the  challenge. 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Location:  Cemetery,  two-story  office  building,  street  between  business  1  and  the 
church 

h.  Considerations:  Eliminate  grass  as  a  factor  (short  grass). 

i.  Analysis  will  be  descriptive/exploratory. 

2.  Marketplace:  Run  a  busy  street  scene,  one-factor  design  in  three  levels 

a.  Goal:  To  establish  a  baseline  for  a  busy  marketplace  route. 

b.  Factors:  Scene 

(1)  Clear,  no  obstacles  (or  to  the  side) 

(2)  Static  obstacles 

(3)  Static  and  dynamic  obstacles  (people  on  controlled  routes) 

c.  Conduct:  Have  the  robot  proceed  through  the  main  street. 

d.  Responses 

(1)  Time  to  achieve  route 

(2)  Obstacle  challenges  met 

(3)  Obstacle  challenges  overcome 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Location:  Center  north/south  run 

h.  Considerations 
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(1)  Some  obstacles  can  be  suspended 

(2)  OOI  will  be  present  and  named  ahead  of  time 

(3)  360°  video  capture  of  the  runs  by  robot 

(4)  Video  capture  of  the  runs  by  CACTF 
i.  Analysis 

( 1 )  Primarily  descriptive . 

(2)  Individuals  from  a  three-person  analyst  team  will  use  a  Likert  scale  to  score 
the  visibility  of  specific  OOI  along  the  route. 

3.  Building  mapping:  Mapping  a  building  can  be  explored  in  a  two-factor  2><3 
factorial  design. 

a.  Goal:  To  establish  a  baseline  for  interior  mapping  capabilities. 

b.  Factors 

(1)  Obstacles  (clear,  room  obstacles) 

(2)  Buildings  (three  buildings) 

c.  Conduct:  Have  the  robot  proceed  through  a  building,  and  map  the  location  of 
walls  and  furniture  (if  it  sees  furniture). 

d.  Responses:  A  number  of  fiducial  marker  positions  will  be  compared  with  actual 
locations  in  a  post-run  analysis  to  determine  offset  distances;  letters  placed  at 
different  heights  and  of  different  heights  will  be  scattered  in  the  area  to  provide  a 
camera  baseline. 

e.  Replications:  (2) 

f.  Randomization:  TBD 

g.  Location:  Police  station,  restaurant,  second- floor  two-story  office  building 

h.  Considerations 

(1)  Some  obstacles  can  be  suspended 

(2)  OOI,  including  people,  will  be  present  and  named  ahead  of  time 

(3)  360°  video  capture  of  the  runs  by  robot 

(4)  Video  capture  of  run  by  CACTF 

i.  Analysis 


14 


( 1 )  Primarily  descriptive . 

(2)  Individuals  from  a  three-person  analyst  team  will  use  a  Likert  scale  to  score 
the  visibility  of  specific  OOI  along  the  route. 

9.  Robotics  Collaborative  Technology  Alliance  Resources 


RCTA  will  provide  the  following: 

•  Two  Talons  with  control  stations  and  operators 

•  Communications/radios  to  operate  in  the  CACTF 

•  Safety  officers  as  required 

•  Differential  GPS  (portable) 

•  Formal  definition  of  mapping  a  room  or  area  (partial  response  already  provided) 

•  Identification  of  all  failure  modes  to  prescribe  appropriate  responses  during  run  (working 
on  this  issue) 

•  Automated  data  collection  to  support  responses  in  experiment  vignettes 

o  Coordinates  of  the  start  point,  end  point,  and  any  waypoints  for  any  plan  generated 
during  any  run 

o  Video  file  from  the  three  cameras  for  runs  involving  OOI  identification 

o  File  containing  Talon  coordinates  (x,y,z,  roll,  pitch,  azimuth)  vs.  time  vs.  status 
(moving,  stopped,  etc.),  vs.  vehicle  direction  (forward  or  backward) 

o  A  file  including  experimental  conditions,  date,  time,  run  number,  etc. 

o  A  map  on  mapping  missions 


10.  Other  Topics 


This  section  lists  a  few  general  topics  for  discussion. 

•  Do  we  need  some  kind  of  periodic  calibration?  Data  collectors  will  record  start  and  end 
points  on  each  run  using  a  portable  GPS  unit.  (There  will  be  a  calibration  prior  to  each  run. 
The  RCTA  will  provide  the  timing  and  method.) 
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•  Can  we  lock  and  report  the  values  of  key  parameters?  (Agreed  to  not  vary  parameters 
within  vignettes.) 

•  Should  key  parameters  be  varied  within  the  design  other  than  what  was  mentioned  above? 
(Some  exploratory  varying  of  parameters  at  the  end  of  the  test  may  be  tried  for  certain 
vignettes.) 

•  Are  there  other  responses  that  could  be  considered?  (Holding  with  these.) 

•  We  need  to  tighten  the  failure  mode.  Time  was  mentioned,  but  what  about  retries  within  a 
run?  Should  there  be  some  repositioning  opportunity  and  retry  for  some  of  the  tests?  (No 
interventions  in  the  initial  baseline  test.  The  Talon  will  succeed  or  fail  on  its  own. 
However,  some  excursions  may  be  run  where  interventions  and  parameter  changes  are 
permitted.)  (I  recommend  allowing  15  excursions  during  the  test  at  the  request  of  the 
developers  and  with  approval  of  the  test  director.  A  maximum  of  four  will  be  allowed  for 
any  one  vignette.  Prior  to  moving  to  a  new  vignette,  but  at  the  conclusion  of  the  current 
vignette,  the  developer  can  call  for  excursion  or  exploratory  runs  [analogous  to  throwing  a 
football  challenge  flag]  where  they  feel  there  is  information  to  be  gleaned  from  a  change  in 
parameters,  intervention,  etc.  By  incorporating  the  challenges  in  the  design  as  opposed  to 
waiting  to  the  very  end,  there  will  be  better  flexibility  to  pursue  important  questions  when 
they  arise.) 

•  When  is  a  run  invalid  and  a  rerun  appropriate?  (Admin  stops  and  reruns  will  be  allowed  in 
certain  situations  [e.g.,  electronic  or  mechanical  failure,  operator  error  in  setup 
configuration,  serious  loss  of  comms,  inadvertent  interloper  on  the  course,  failure  to  collect 
files  with  robot  data/video].) 

•  We  need  two  Talons  for  this  effort.  Both  will  be  used  fully  in  the  plan.  However,  if  one 
fails,  we  will  have  to  prioritize  on  the  fly  what  portions  of  the  remaining  experiment  will  be 
performed.  (We  will  have  two  Talons.) 

•  Who  is  responsible  for  test  setup  items  (static  obstacles  [cones,  vehicles,  barrels, 
mannequins],  plywood  for  door  widths,  platform  and  ramp,  blocks/bricks  to  adjust  heights 
of  platforms)?  (GDRS  will  handle  this.) 

•  Do  we  want  to  add  something  like  bags  of  mulch  as  steps  for  the  robot  to  climb  or 
descend?  (No.) 

•  What  operating  conditions  would  lead  to  an  e-stop  (e.g.,  too  big  of  a  drop-off,  too  much  of 
a  roll)?  (A  safety  line  will  be  used  to  protect  the  robot  from  damage  in  runs  pushing  the 
operating  limits.) 
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Appendix.  Runs  and  Locations 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Surfaces  Vignette 


A 


Runs  1-2  (Low  Grass  x  Level) 


Runs  3-8  (Low  Grass  x  [uphill,  downhill,  side  hill]) 
Runs  9-14  (Gravel  x  [uphill,  downhill,  side  hill]) 


Runs  17-24  (Pavement  x  [uphill,  downhill,  side  hill,  level]) 


Notes 

•  Alpha  Team 

•  Grass  cut  on  approach  south  of  church 

•  32  (28)  Scheduled  runs,  randomized  within  location,  except 
side  hill  runs  which  come  last;  1  side  hill  run  per  location 

•  Excursions  (maximum  4) 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Cul  de  Sac  Vignette 


Notes 

•  Bravo  Team 

•  Grass  cut  in  cemetery  and  out 

•  Severity  reflects  full  interaction  with  cul  de  sac  or  glancing  blow 

•  Maze  severity  will  be  based  on  obstacle  density  and  the 
directness  of  a  path  through  (diagram  below) 

•  18  Scheduled  runs,  randomized  within  location,  except 
the  maze  which  follows  easy  to  hard,  2  reps  each 

•  Corridor  setting  wide,  except  for  maze  which  should  be  maze  width 

•  Excursions  (maximum  4) 


End 


End 


End 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Alley  Vignette 


Notes 

•  Bravo  Team 

•  Enter  alley  from  the  street 

•  Vary  narrowness  of  alley  using  plywood  on  the 
north  side  of  alley  to  full  alley  (no  plywood)  and 
two  other  widths  TBD 

•  6  Scheduled  runs,  sequence  wide  to  narrow  alley,  2  reps  each 

•  Excursions  (maximum  4) 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Doorway  Vignette 


Runs  57-92  (3  Door  widths  x  3  Angles  of  approach  x  Door 
Ajar/Not) 


Notes 

•  Bravo  Team 

•  Enter  door  from  patio  area  attached  to  south 
end  of  police  station. 

•  Vary  door  width  with  plywood  and  angle  of 
approach  by  initial  position.  Door  ajar  at  60 
degree  open  or  not. 

•  36  Scheduled  runs,  randomized  across  factors,  but  within  a  rep 

•  Excursions  (maximum  4) 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
People  Vignette 


Runs  93-112  (Path  x  Speed) 


Runs  93-112  (Path  x  Speed) 


to 

to 


Runs  93-112  (Path  x  Speed) 


Runs  93-112  (Path  x  Speed) 


Notes 

•Alpha  Team 

•  Two  baseline  runs  (no  people)  from  the  surfaces  vignette 

•  Paths  (yellow  [walk],  yellow  continuation  white  [jog]) 

•  Proximity  of  movers  to  TALON  (3-4  m) 

•  Orbit  path  in  same  direction  as  sensor  rotation 

•  Corridor  setting  to  keep  TALON  on  street 

•  20  Scheduled  runs,  randomized 

•  Excursions  (maximum  4) 


Runs  93-112  (Path  x  Speed) 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Building  Mapping  Vignette 


Runs  113-120  (Police  station  (left)  and  2-story  Office  Bldg  (right, 
2nd  floor)  x  obstacles) 


Notes 

•  Bravo  Team 

•  Obstacle  condition  (clear,  furniture  +  OOI) 

•  360  Video  captured  for  later  analyst  review  in  identifying  OOI 

•  A  class  of  OOI  will  be  numbers  on  the  wall  to  establish  a  baseline 
on  the  camera. 

•  12  Scheduled  runs,  2  reps  after  each  set-up/location 

•  Excursions  (maximum  4) 


Runs  121-124  (Restaurant  x  obstacles) 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Drop-off  Vignette 


Runs  125-136  (Level  Concrete  x  3  Heights  x  2  Landings) 


Run  El  (Excursion  [angled  approach  to  drop-off]) 


to 

4^ 


Runs  137-148  (Side  Hill  Grass  x  3  Heights  x  2  Landings) 


Runs  149-160  (Level  Grass  x  3  Heights  x  2  Landings) 


Notes 

•  Alpha  Team 

•  Heights:  15",  10",  5" 

•  Landings:  plywood,  sod 

•  Need  to  have  the  door  open  and  start  from  within  Business  1 

•  36  Scheduled  runs,  sequenced  within  each  location,  varying  height 
(5"  to  15")  with  plywood  first,  then  sod,  2  reps  each 

•  Excursions  (maximum  4) 

•  Excursion  run  conditions  (Height  &  Landing)  TBD 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Market  Place  Vignette 


Runs  161-170  (Market  Place  with  [clear,  static  obstacles,  static 
and  dynamic  obstacles]) 


Notes 

to  -Bravo  Team 

^  -Two  baseline  runs  with  no  obstacles  and  then  4  each  with 

static  or  static  &  dynamic 

•  Paths  of  dynamic  obstacles  (people,  vehicles)  TBD 

•  Dense  collection  of  obstacles  (cones,  barrels,  vehicles) 

•  360  Video  will  be  recorded  for  later  OOI  identification  by 
analysts 

•  Ideal  opportunity  for  video  from  the  hotel  window 

•  10  Scheduled  runs,  reps  after  each  set-up 

•  Excursions  (maximum  4) 


RCTA  2011  Baseline  Assessment 
Appendix  A:  Runs  &  Locations 
Stairs  Vignette 


1  Step 


to 

as 


» 


Notes 

•  Alpha  &  Bravo  Teams 

•Riser  (4",  8",  12");  Tread  (8",  12",  24");  Run  (96"); 

11  combinations 

•  To  scale  (1/4"  =  4",  except  96"  run,  TALON  approx.) 

•  Riser  is  open  (e.g.,  fire  escape)  or  closed 

•  Townhouse  steps  (~6  runs  if  feasible,  TBD) 

•  ARL  Facility  (barn)  for  remainder  of  runs 

•  Safety  line  to  prevent  rolling  backwards 

•  Red  signifies  nominal  climb 

•  44  Scheduled  runs,  1  step  first,  2  step  ordered 
by  riser, open/closed,  then  tread 

•  Excursions  (maximum  4) 


NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BEL  VOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  HRR 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  MT 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  D 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 
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4  GENERAL  DYNAMICS  ROBOTICS  SYS 
B  STUART 
J  GONZALEZ 
R  DEAN 
R  MITCHEL 
1234  TECH  CT 
WESTMINSTER  MD  21 157 

2  ROBOTIC  RSRCH  LLC 
A  LACAZE 

K  MURPHY 
STE  300 

555  QUINCE  ORCHARD  RD 
GAITHERSBURG  MD  20878 

1  NASA  GLENN 
RDRL  VT 
M  J  VALCO 
BLDG  21  RM  138 
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1  ADELPHI  LAB  CTR 
RDRL  CII 
B  D  BROOME 
BLDG  205  RM  4E025D 
ADELPHI  MD  20783-1 197 
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M  THOMAS 
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K  COSENZO  (3  CPS) 

RDRL  VTA 
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H  EDGE  (3  CPS) 

M  CHILDERS  (3  CPS) 
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